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Recommendation   #1 

SDB   reexamine  its  management  control 

system  to  assure  adequate  management 
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rent year  dollars.  13 

Bureau   Reply:      Do  not  concur.      See  page  41 . 
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CHAPTER   I 

INTRODUCTION 

During  late  1982,  the  Office  of  the  Legislative  Auditor  con- 
ducted a  survey  of  state  data  processing  activities.  One  of  the 
potential  audit  areas  identified  in  the  survey  was  design  and 
development  of  computer  software  systems  by  the  Systems  Develop- 
rnent  Bureau,  Information  Services  Division.  The  Legislative  Audit 
Committee  requested  an  audit  of  this  area.  This  report  results 
from  our  audit. 

OBJECTIVES   OF  AUDIT 

The  primary  objective  of  the  audit  was  to  determine  whether 
the  management  of  the  Systems  Development  Bureau  (SDB)  was 
fostering  well-designed  and  controlled  computer  systems.  This 
included  determining: 

1.  The  existence  of  sufficient  management  controls  within  the 
current  systems  development  process. 

2.  The  adequacy  of  management  controls  over  maintenance  and 
changes  to  software. 

3.  The  reasonableness  of  costs  allocated  to  SDB. 

4.  The  reasonableness  of  SDB's  contract  provisions. 

5.  The  adequacy  of  SDB's  training  program  and  evaluation 
process. 

6.  The  adequacy  of  the  current  method  of  priority  setting  for 
projects. 

7.  Compliance  with  applicable  laws,    rules  and   regulations. 

8.  The  adequacy  of  controls  to  detect  unauthorized  computer  use 
and  to  prevent  or  detect  unauthorized  file  accesses  or 
changes. 

9.  The  adequacy  of  controls  over  consultant  contracts. 


During     the     audit     we     asked     SDB     management     for     written 
responses     to     selected     audit     points.        These     points     related     to 


potential   report   issues  and   recommendations,   and   infor-med  manage- 
ment of  tinese  issues  during   the  audit. 

SCOPE  OF  AUDIT 

We  concentrated  on  SDB's  management  of  software  development 
and  maintenance.      In  addition,   vje  examined: 

— Cost  effectiveness  of  in-house  staff  vs.    consultants   for  devel- 
opment. 

— SDB   funding  and  expenditures. 

— Personnel  administration  at  SDB. 

— Tlie  priority  setting  mecPianism  for  SDB   projects. 

— Compliance  with  laws,    rules,   and  policies. 

— Controls  over  SDB  computer  use. 

As  part  of  our  audit  we  reviewed  in  detail  the  documentation 
of  four  major  projects  which  were  developed  by  SDB,  We  also 
examined  the  documentation  pertaining  to  four  systems  SDB  main- 
tains. We  examined  SDB/agency  contracts.  Personnel  files  were 
reviewed  and  SDB's  training  program  was  examined.  We  inter- 
viewed SDB  staff  and  agency  staff  concerning  development  and 
maintenance  of  software  systems. 

The  audit  focused  on  the  managerial  operation  of  SDB  and 
was  conducted  in  accordance  with  generally  accepted  governmental 
performance  auditing   standards. 

Management  Memoranda 

As  a  result  of  the  preliminary  survey,  we  sent  a  management 
memorandum  to  SDB  officials.  We  noted  programs  stored  on  the 
computer  which  indicated  possible  unauthorized  computer  use.  We 
suggested  a  periodic  review  be  done  to  reduce  the  potential  for 
unauthorized  computer  use. 

The  audit  identified  additional  issues  for  which  management 
memoranda  were  sent.      The  issues  included: 


1.  Limiting  access  to  programs  stored   in   the  computer. 

2.  Limiting   access  to  various  SDB   computer  files. 

3.  Providing  more  training   for  analysts. 

4.  Having  the  responsible  analysts  evaluate  the  work  of  the 
programmers. 

5.  Conducting  routine  background  checks  on  prospective 
employees. 

6.  Creating  the  documentation  pertaining  to  a  change  in  a  pro- 
gram as  required  by  the  bureau's  Administrative  and 
Technical  Manual. 


These   issues   did   not   merit  detailed   discussion    in   this   report. 
Management  has  been   informed  and   has  taken  corrective  action. 


CHAPTER   II 

BACKGROUND 

The  Systems  Development  Bureau's  (SDB)  primary  function  is 
to  design,  develop,  and  maintain  computer  software  systems  for 
state  agencies.  The  Systems  Development  Bureau  is  part  of  the 
Information  Services  Division  (formerly  Computer  Services  Division) 
of  the  Department  of  Administration. 

SDB  ORGANIZATION 

The  Information  Services  Division  recently  reorganized.  The 
following  illustration  portrays  SDB's  current  organization.  At  the 
time  of  our  audit,  the  Office  Automation  Center  was  part  of  SDB 
and  the  Database  Design  and  Administration  Section  was  part  of 
another  bureau.  Under  the  reorganization,  the  Office  Automation 
Center  has  become  a  separate  bureau. 

ORGANIZATION  CHART 
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STAFFING 

At  the  time  of  the  audit,  the  Systems  Development  Bureau 
was  allocated  21  FTE.  The  total  included  the  bureau  chief,  15 
professional  development  staff  (programmers  and/or  analysts),  4 
Office  Automation  Center  staff,   and   1    FTE  for  secretarial   staff. 

FUNDING 

There  is  a  single  internal  service  fund  for  the  Information 
Services  Division.  During  the  audit,  revenues  and  expenditures 
for  SDB,  the  Office  Automation  Center,  Records  Management 
Bureau,  and  the  central  computer  operation  were  handled  through 
the  internal  service  fund.  The  Office  Automation  Center  was 
funded  through  computer  processing  charges.  The  rest  of  SDB 
was  funded  by  revenues  generated  through  providing  development 
and  maintenance  services  to  user  agencies. 

The  following  chart  shows  income  and  expenditures  for  the 
Information  Services  Division  (the  forerunner  of  SDB)  and  SDB  for 
the  five  fiscal  years  1979-80  through   1983-8U. 

ISD/SDB   Income   and  Expenditures    (Unaudited) 

ISD  ISD  ISD  SDB  SDB 

Item  FY   79-80        FY   80-81        FY   81-82        FY   82-83  FY   83-8A 


Income   *  $669,112        $736,983        $600,588        $529,551  $657,188 

Expenses**  638,810  702,338  677,215  521,317***        606,531*** 

Difference        $   30,302        $    34,645        $(76,627)      $     8,234  $   50,657 


*  Income,    Revenue   and   Prior  Year   Income/Revenue   Adjustments. 

**        Expenses,    Withdrawals,    and   Prior-Year   Expenses/Withdrawals. 
***      Understated     by     $38,530     and      $59,266,      respectively,     because 

Computer    Processing    billings     and    Computer     Services    Division 

overhead   are   not    included. 

^   Source:      Montana  Financial  Reports   FY   79-80,   80-81,   and  81-82. 

2 

Source:   Computer  Services  Division  Presentations  to  Legislative 

Finance  Committee. 
Source:   Statewide  Budgeting  and  Accounting  System. 
Income  was  down  during  FY  82-83  due  to  fewer  projects. 
Illustration  1 
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BUREAU   FUNCTIONS 

SDB  has  two  primary  functions:  1)  development  of  computer 
software  systems,  and  2)  maintenance  of  computer  systems.  These 
two  areas  are  discussed  below. 

Development 

Systems  Development  Bureau  designs,  develops,  and  installs 
computer  software  systems  for  state  agencies  requesting  its  ser- 
vices. SDB  has  created  systems  for  such  agencies  as  Secretary  of 
State,  Public  Employees'  Retirement  Division,  Central  Payroll 
Division,  Department  of  Labor  and  Industry,  Department  of  Health 
and  Environmental   Sciences,   and   Purchasing   Division. 

SDB  uses  a  three  phase  methodology  in  the  design,  develop- 
ment, and  implementation  of  an  automated  system.  The  three 
phases  include:  1)  functional  specifications;  2)  systems  design; 
and  3)   system  test  and  installation. 

The  functional  specifications  phase  produces  a  statement  of 
the  agency's  information  needs  and  appropriate  solution(s).  The 
purpose  of  this  phase  is  to  provide  the  agencies  with  an  overview 
of  the  proposed  systen(s)  so  they  can  determine  whether  the 
system(s)  will  meet  their  business  requirements.  More  than  one 
alternative  can  be  provided.  If  the  agency  does  not  concur  with 
the  system(s)   proposed,   the  work  stops. 

Systems  design  and  development  is  the  second  phase.  During 
this  phase,  the  specifications  arc  developed  into  a  computer 
system.  Then,  the  computer  programs  are  coded.  At  the  end  of 
the  second  phase,  a  functioning  computer  system  has  been  devel- 
oped. 

System  tests  and  installation  is  the  third  and  final  phase. 
During  this  phase,  the  agency  tests  the  system  to  see  that  it  is 
functioning  properly  and  meets  the  agency's  needs.  SDB  will  also 
help  with  installation  of  the  system  for  the  agency. 

Maintenance 

SDB  also  enters  into  software  maintenance  agreements  with 
state      agencies.        Those      agreements      provide      for      consultation. 


assistance  when  the  system  fails  to  operate  properly,  and  system 
enhancement/maintenance.  Monthly  payments  for  the  agreements 
are  based  upon  established  rates  and  work  actually  performed 
(time  and  materials).  Currently  the  charge  is  $29  per  hour. 
Agencies  for  which  SDB  maintains  systems  include  Department  of 
Fish,  Wildlife,  and  Parks;  Department  of  Agriculture;  Secretary  of 
State;   and  State  Auditor's  Office. 

Maintenance  of  the  Statewide  Budgeting  and  Accounting 
System  (SBAS)  requires  two  SDB  employees.  Another  SDB 
employee  works  full-time  maintaining  the  Payroll/ Position 
Control/Personnel   System. 

Workload 

The  following  chart  summarizes  the  workload  for  the  bureau. 

SDB  WORKLOAD 

Item  Number 

Current  Development  Contracts  15 

Development  Contracts  Completed  FY  81-82  "       6 

Development  Contracts  Completed  Fiscal  Year  82-83  10 

Current  Software  Maintenance  Contracts  23 

Cost  Estimates  for  1981  Session  5 

Cost  Estimates  for  1983  Session  4 

Source:   Systems  Development  Bureau 

Illustration  2 

COMPLIANCE 

As  part  of  our  audit  we  reviewed  compliance  with  laws, 
administrative  rules,  and  policies  relating  to  the  bureau  and  its 
activities.  We  examined  compliance  with  statutes  as  they  applied  to 
our  audit  period.     We  found  the  bureau  to  be  in  compliance. 

For  those  rules  and  regulations  not  related  to  the  adminis- 
tration of  the  bureau,  and  therefore  not  tested  for  compliance, 
nothing  came  to  our  attention  during  the  audit  that  indicated 
significant  non-compliance. 


CHAPTER   III 
MANAGEMENT  CONTROLS 

In  our  audit,  we  focused  on  the  management  control  system 
SDB  has  developed  and  implemented  to  help  assure  quality 
computer  systems  are  delivered  to  customer  agencies.  A  good 
management  control  system  helps  assure  efficient  and  effective 
operations.  Management  controls  also  assure  adequate  procedures 
are  followed. 

We  found  SDB's  development  and  maintenance  process,  as 
stated  in  its  procedures  manual,  contains  numerous  management 
controls.  We  compared  SDB's  stated  controls  with  those  suggested 
in  Control  Objectives  (EDP  Auditors  Foundation)  and  Computer 
Control  and  Audit  (Mair,  Wood,  and  Davis).  Our  analysis 
indicated  that  certain   important  controls  did  not  exist. 

Much  of  our  work  centered  on  verifying  that  the  controls  SDB 
said  it  had  actually  existed.  A  control  which  is  not  documented  or 
which  is  not  complied  with  is  usually  ineffective.  In  several 
cases,  we  could  not  verify  the  existence  of  the  control  because  of 
insufficient  documentation.  For  other  controls,  the  documentation 
we  reviewed  indicated  non-compliance. 

From  our  analysis,  we  are  unable  to  conclude  SDB  has 
adequate  management  controls  to  assure  agencies  receive  quality 
computer  systems.  We  recommend  SDB  reexamine  its  management 
control  system  to  assure  adequate  management  controls  exist,  to 
assure  the  controls  are  verifiable  through  documentation,  and  to 
assure  the  control  procedures  are  followed.  Chapter  IV  and 
Chapter  V  discuss  our  specific  suggestions  for  improved  manage- 
ment controls. 

RECOMMENDATION  #1 

WE   RECOMMEND   SDB  REEXAMINE   ITS   MANAGEMENT 

CONTROL  SYSTEM  TO  ASSURE  ADEQUATE  MANAGEMENT 

CONTROLS  EXIST,  THE  CONTROLS  CAN  BE  VERIFIED  BY 

DOCUMENTATION,  AND  THE  CONTROL  PROCEDURES  ARE 

FOLLOWED. 


CHAPTER  IV 

SYSTEMS  DEVELOPMENT  PROCESS 

The  systems  development  process  incorporates  the  study  of 
existing  systems,  agency  needs,  and  design  of  new  or  improved 
methods  to  meet  identified  needs.  This  process  is  initiated  by  an 
agency  contacting  Systems  Development  Bureau  (SDB)  to  discuss  a 
software  system.  SDB  assigns  an  analyst  to  the  project.  The 
analyst  goes  into  the  agency  to  learn  the  current  system  and 
identifies  the  agency's  information  needs.  The  analyst  also 
develops  cost  projections  detailing: 

— projected    costs   of   the   current   operation    for   the   next   four   to 
five  years; 

— cost  estimates   for   the  proposed   system's  development,   conver- 
sion,  and   installation;   and 

— the   proposed    system's    projected    operational    costs    for   four   to 
five  years. 


The  analyst  incorporates  all  the  information  into  a  specifications 
document. 

After  the  agency  reviews  and  approves  the  specifications 
document,  SDB  identifies  in  detail  the  required  data  files  and 
computer  interrelationships  for  all  processing  needs  (daily, 
monthly,  quarterly,  annual  reporting,  etc.).  The  individual 
programs  comprising  the  system  are  then  designed  and  developed 
(coded).  Each  program  or  group  of  programs  is  tested  to  deter- 
mine if  any  errors  were  made  in  the  coding.  The  programs  may 
then  be  put  together  in  groups,  and  may  be  tested  again  to  deter- 
mine if  any  errors  are  present. 

The  agency  reviews  the  tested  programs  and  conducts  their 
own  tests,  using  their  own  test  data.  SDB  again  corrects  any 
errors  discovered.  The  responsibility  for  the  system  is  then 
transferred  to  the  agency.  A  maintenance  agreement  between  SDB 
and  the  agency  is  created  so  SDB  can  provide  any  necessary 
assistance  to  the  agency  after  implementation.  (Maintenance  agree- 
ments are  discussed   in  Chapter  V). 


During  the  audit  we  reviewed  the  systems  development  docu- 
mentation for  four  systems.  Three  of  the  systems  were 
implemented  since  SDB  was  formed.  The  fourth  was  still  in  the 
development  process.  We  compared  the  development  process  as 
evidenced  by  SDB's  documentation  with  a  well  controlled 
development  process  as  described  in  Control  Objectives  (EDP 
Auditors  Foundation)  and  Computer  Control  and  Audit  (Mair, 
Wood,  and  Davis).  We  developed  audit  findings  related 
to:  design  process;  testing;  documentation;  and  review,  planning, 
and  post-implementation  review.  These  areas  are  discussed  in 
detail  below. 

DESIGN   PROCESS 

The  design  process  should  have  sufficient  controls  to  assure 
the  agency  that  the  suggested  alternative(s)  will  meet  the  agency's 
information  needs.  In  addition,  the  agency  should  be  given 
complete  and  accurate  information  to  make  a  decision.  Our  audit 
findings  included:  contents  of  design  documents;  design  alterna- 
tives;  quantification  of  benefits;   and  present  value  anafysis. 

Controls  Descriptions  in  Design   Documents 

We  compared  SDB's  design  document  contents  with  those 
suggested  by  the  EDP  Auditors  Foundation  in  Control  Objectives. 
In  reviewing  the  design  documents,  we  noted  some  documents 
lacked  pertinent  information  on  the  controls  to  be  built  into  the 
system. 

The  following  illustration  shows  our  evaluation  of  the  descrip- 
tion of  controls  in  four  systems  reviewed. 
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The  chart  indicates  a  lack  of  adequate  discussion  of  controls 
in   the  design   documents  for  each  of  the  systems. 

Designing  a  computer  system  involves  detailing  not  only  the 
processing  required  to  meet  the  agency's  information  needs,  but 
also  the  measures  to  be  taken  to  assure  adequate  security  and 
integrity  for  the  system.  The  design  document  should  include  the 
measures  to  be  taken  concerning  security  and  integrity  so  the 
agency  can  assess  the  adequacy  of  controls. 

RECOMMENDATION    #2 

WE    RECOMMEND    SDB    REQUIRE   A    DETAILED    DISCUSSION    OF 

SYSTEM   CONTROLS    IN   THE   DESIGN    DOCUMENTS. 


Analysis  of  Alternatives 

SDB  can  become  involved  in  a  systems  development  project  at 
two  points.  Sometimes  an  agency  contacts  SDB  to  develop  a 
computer  system  after  the  agency  has  performed  some  analysis  and 
has  decided  on  a  design.  In  other  cases,  the  agency  recognizes 
an  information  system  problem  and  contacts  SDB  for  assistance  in 
deciding   how  best  to  solve  the  problem. 

We  examined  SDB's  procedures  for  addressing  these  latter 
situations.  Specifically,  we  concentrated  on  the  information  pro- 
vided to  agency  management  upon  which  to  make  a  decision.  The 
following   sections  discuss  our  suggestions  for  better  information. 


Design  Alternatives 

When  making  a  decision  about  design  of  a  computer  system, 
the  two  major  criteria  are  performance  and  cost.  Since  there  are 
usually  several  ways  to  solve  an  agency's  information  needs, 
agency  management  must  balance  cost  and  performance  to  choose 
the  proper  alternative. 

Current  design  documents  do  not  discuss  alternatives  for  the 
design  of  the  system.  The  analyst  developing  the  system  presents 
what  he  believes  to  be  the  preferred  solution  to  the  agency's 
needs. 

Agencies  should  be  formally  presented  with  alternatives  in  the 
design  of  the  system.  Without  the  presentation  of  alternatives  and 
the  estimated  costs  for  each  option,  the  agency  cannot  make  an 
informed  decision  concerning  the  design  and  development  of  a 
system. 

For  example,  an  agency  might  choose  a  less  technically 
advanced  system  over  a  "state-of-the-art"  system.  However,  the 
current  procedures  might  not  offer  the  agency  this  choice  because 
the  analyst  believes  the  "state-of-the-art"  system  is. the  correct 
solution. 

Agency  managers  should  make  their  decisions  based  on  per- 
formance and  cost.  SDB  role  is  only  to  suggest  the  technically 
preferred  alternatives.  Thus,  an  agency  may  accept  a  system 
which  does  not  do  quite  as  much  if  the  cost  of  the  development 
and  operations  can  be  significantly  decreased. 

RECOMMENDATION   #3 


WE  RECOMMEND  SDB  FORMALLY  PRESENT  AGENCIES  WITH 
ALTERNATIVES  AND  THE  ASSOCIATED  COSTS  SO  AGENCY 
MANAGEMENT  HAS  MORE  INFORMATION  TO  USE  IN  MAKING 
DECISIONS. 


Quantification  of  Benefits 

The   benefits   of   the   computer    software    system    to    the   agency 
are      not      quantified      in      the      design      documents.        Anticipated 
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administrative  benefits  are  specified,  but  for  three  of  the  four 
systems  reviewed,  benefits  were  not  quantified.  Not  all  benefits 
can  be  quantified  in  dollars  but  many  can  have  numbers  attached 
covering  decreased  response  time  and  increased  documents 
processed.  While  some  benefits  may  not  be  quantifiable,  those 
which  are  should  be  presented  so  agency  management  can  make  a 
better  informed  decision  concerning  the  most  beneficial  system. 
The  quantification  of  costs  and  benefits  also  allows  the  objective 
evaluation  of  development  activities  after  the  project  has  been 
completed. 

RECOMMENDATION   §H 

WE     RECOMMEND     SDB     REQUIRE     ANALYSTS     TO     QUANTIFY 

BENEFITS   IN   THE  DESIGN   DOCUMENTS  WHERE   POSSIBLE. 


Present  Value  Analysis 

Analysts  project  the  costs  of  a  system  to  an  agency  for  the 
next  four  or  five  years  in  the  design  document.  Currently  SDB's 
analysis  does  not  take  into  account  future  dollars  are  worth  less 
than  current  dollars.  A  technique  called  present  value  analysis 
allows  cost  to  be  presented  in  current  year  dollars.  Without  such 
an  analysis  the  agency  does  not  know  the  true  costs  of  a  system 
over  a  period  of  years. 

RECOMMENDATION   #5 


WE      RECOMMEND      COST      PROJECTIONS      BE      STATED      IN 
CURRENT  YEAR  DOLLARS. 


TESTING 

The  testing  phase  catches  flaws  which  may  have  appeared  in 
a  system  during  design  and  development.  Adequate  testing  proce- 
dures assure  the  customer  the  product  will  meet  the  customer's 
needs.  We  examined  SDB's  program  and  system  testing  standards 
and   actual    testing    practices   as    evidenced    from    systems    completed 


in    the    last    18    months.      We    developed    findings    related    to    testing 
standards  and  the  system  acceptance  test. 

Testing  Standards 

We  reviewed  standards  SDB  established  for  testing  of  systems 
developed  or  subjected  to  major  modifications.  We  found  SDB's 
procedures  manual  states  programs  should  be  tested  but  does  not 
have  specific  standards  for  testing  individual  programs  or  an 
entire  software  system. 

The  EDP  Auditor's  Foundation  in  Control  Objectives  suggests 
specific  testing  standards.  The  standards  should  require  a 
written  testing  plan  and  a  review  of  the  plan  by  appropriate 
supervisory  personnel.  Appropriate  SDB  and  agency  management 
should  approve  the  plan  for  the  test  of  the  system  as  a  whole. 
The  testing  standards  should  require  that  the  plan  include  what  is 
to  be  tested,  how  it  will  be  tested  and  what  will  be  considered 
acceptable  results.  The  same  people  who  approved  the  testing 
plans  at  each  stage  should  also  review  the  test  results  before 
passing  it  on  for  the  next  level  of  testing  or  approving  the  system 
for  implementation.  The  standards  should  require  extensive 
retesting  if  major  flaws  are  noted. 

RECOMMENDATION    #6 

WE  RECOMMEND  SDB  DEVELOP  AND  IMPLEMENT  SPECIFIC 
TESTING  STANDARDS,  INCLUDING  REVIEW  AND  APPROVAL 
OF  TEST   PLANS  AND   TEST   RESULTS. 


Systems/Acceptance  Test 

A  computer  system  is  usually  made  up  of  several  programs. 
Each  program  performs  a  different  but  related  task.  A  useful 
analogy  is  to  think  of  the  programs  as  gears  in  a  machine.  The 
individual  gears  may  be  properly  formed  but  may  not  mesh  well 
causing  the  machine  to  malfunction.  The  same  thing  can  happen 
with  computer  systems.  The  individual  computer  programs  can 
work  well  but  the  interaction  of  the  programs  can  cause  problems. 
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SDB's  development  process  usually  calls  for  the  completion  of 
small  groups  of  related  programs  (modules)  at  varying  times.  SDB 
tests  the  modules  as  completed  and  transfers  the  modules  indi- 
vidually or  in  small  groups  to  the  agency  for  testing.  The  agency 
is  to  test  each  module  and  accept  the  module  or  return  it  to  SDB. 
We  did  not  find  evidence  all  of  the  modules,  plus  the  associated 
manual  procedures,   are  tested  as  a  system  prior  to  implementation. 

We  reviewed  in  detail  one  system  that  was  transferred  in  such 
a  manner.  Fourteen  months  after  the  transfer,  the  agency  was 
still  paying  SDB  to  fix  the  system  so  it  would  operate  the  way 
originally  intended.  In  the  first  four  months  of  use  we  identified 
$8,000  SDB  billed  the  agency  to  fix  problems.  Another  $10,000 
was  billed  by  SDB  in  the  next  ten  months  to  fix  items.  About 
$5,000  was  spent  for  computer  charges  in  conjunction  with  the 
above  amounts.  SDB  officials  noted  that  they  used  this  method 
because  the  agency  wanted  the  system  put  into  production  as  soon 
as  possible. 

A  well  controlled  systems  development  process  requires  a 
systen/acceptance  test  be  conducted  before  the  system  is  imple- 
mented for  final  usage.  A  comprehensive  systems/acceptance  test 
is  one  way  to  detect  and  correct  deficiencies  in  a  system  before  it 
is        implemented        for       use       by        the       agency.  Without       a 

system/acceptance  test,  the  agency  will  not  know  if  the  entire 
system  is  operating  correctly. 

We  recommend  SDB  implement  the  following  procedures  to 
improve  the  quality  of  systems  developed: 

1 .  Attempt  to  turn  over  completed  systems  which  have  been 
tested   in  their  entirety  to  agencies  for  acceptance  testing. 

2.  If  modules  are  to  be  completed  separately  and  tested  as 
completed,  test  the  most  recently  completed  module  along  with 
all  related  modules  and  manual  procedures  to  determine  how 
they  interact. 


15 


RECOMMENDATION   #7 

WE  RECOMMEND  SDB  AND  AGENCIES  TEST  ALL  RELATED 
MANUAL  AND  COMPUTERIZED  PORTIONS  OF  THE  SYSTEM 
BEFORE  A  SYSTEM   IS  ACCEPTED  AS  COMPLETE. 


DOCUMENTATION 

Documentation  serves  three  major  purposes.  During  develop- 
ment, documentation  eases  the  transition  should  a  new  person  be 
assigned  to  a  project.  Documentation  also  provides  a  supervisory 
person  with  material  to  review  to  assure  that  a  project  is  proceed- 
ing properly.  When  system  maintenance  becomes  necessary, 
documentation  assists  the  person  performing  the  work,  thereby 
increasing  efficiency.  Maintenance  of  a  major  system  is  a  costly 
undertaking.  The  Statewide  Budgeting  and  Accounting  System 
maintenance  exceeds  $100,000  per  year.  Thus,  making  maintenance 
easier  is  a  significant  consideration. 

Our  review  of  SDB's  development  process  included  a  compari- 
son of  current  documentation  and  standards  with  those  described 
in  Control  Objectives  and  Computer  Audit  and  Control.  The 
following  sections  discuss  our  suggestions  for  improvements  in 
standards,   formal  workpapers,   and  completion  of  documentation. 

Documentation  Standards 

We  reviewed  the  bureau's  standards  for  documentation.  We 
noted  the  standards  do  not  address  some  important  items.  These 
include: 

— when  documentation   should  be  prepared; 

— the    extent    of    documentation    required    for    the    administrative 
aspects  of  the  project   (i.e.,   working   papers); 

— supervisory  review; 

— what  documentation  of  testing   is  required;   and 

— what  the  retention  period  should  be  for  documentation. 


16 


We  noted  documentation  of  these  aspects  of  the  development 
process  is  minimal.  SDB's  work  plan  checklist  asks  for  documenta- 
tion at  various  stages  of  the  process,  but  specific  standards  for 
the  form,  content,  and  detail  of  this  documentation  do  not  exist  in 
many  cases. 

Inadequate  documentation   standards   usually   lead  to: 

1.  Decreased  ability  to  assure  quality  work   is  performed. 

2.  Increased  time  orienting  new  staff  to  projects. 

3.  Increased  future  software  maintenance  costs. 

RECOMMENDATION   #8 

WE   RECOMMEND   SDB    IMPLEMENT   SPECIFIC   DOCUMENTATION 

STANDARDS. 


Completion  of  Documentation 

A  major  control  to  assure  a  quality  system  is  placed  into  final 
use  is  a  requirement  for  all  documentation  to  be  completed  prior  to 
implementation.      No  such   requirement  currently  exists. 

Among  the  items  of  documentation  which  need  to  be  completed 
prior  to  implementation  are: 

1.  system  level  flowcharts  for  the  actual  system  as  developed; 

2.  program  logic  narratives  for  the  actual  programs; 

3.  operating   instructions  for  technical  data   processing   staff;   and 

4.  operating   instructions  for  users  in   the  agency. 

The    first    three    items    are    SDB's    responsibility.      We    did  not 

find    any     review    mechanism    at    SDB     to    assure    these     items  are 

properly  completed.  For  those  development  projects  reviewed,  we 
found  SDB's  documentation  was  not  in  place  at  implementation. 

The     agency     operating      instructions     are     supposed     to  be 

prepared  by  the  agency.  We  found  these  manuals  are  often  not 
prepared  prior  to  implementation. 
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We    recommend    SDB    take   the   following    steps   to   assure   timely 
completion  of  documentation: 

1.  Make  completion  and  review  of  documentation  part  of  SDB's 
testing  process.  SDB  should  not  consider  a  system  complete 
unless  SDB's  documentation   is  complete. 

2.  Request  agencies  provide  SDB  with  an  outline  of  the  proce- 
dures the  agency  will  be  using  before  the  system  or  modules 
are  given  to  the  agency  for  testing.  (This  outline  should 
serve  as  the  basis  for  a  user's  manual.)  Review  the  contents 
of  the  outline  and  provide  the  agency  with  feedback  on  the 
adequacy  of  procedures  so  the  agency  can  finalize  its  user's 
manual. 

3.  SDB  should  also  stress  to  agencies  and  to  their  staff  the 
importance  of  completing  all  documentation  prior  to  implemen- 
tation. 


RECOMMENDATION   #9 
WE   RECOMMEND   SDB: 

A,  INCLUDE  COMPLETION  AND  REVIEW  OF  DOCUMENTA- 
TION   IN   THE  TESTING   PROCESS. 

B.  REQUEST  AGENCIES  PROVIDE  SDB  WITH  AN  OUTLINE 
OF  PROCEDURES  AND  PROVIDE  THE  AGENCIES  WITH 
FEEDBACK   ON   THE  OUTLINE. 


REVIEW,    PLANNING,   AND   POST-IMPLEMENTATION   REVIEW 

Our  comparison  of  SDB's  development  process  disclosed  weak- 
nesses related  to  supervisory  review,  implementation  planning,  and 
post-implementation  review.  The  following  sections  include  our 
recommendations  for  improvements. 

Supervisory  Review 

Supervisory  review  of  work  is  used  as  a  mechanism  for  assur- 
ing consistency  and  quality  of  work.  We  noted  very  little  docu- 
mentation of  supervisory  review  for  SDB.  Most  of  the  documenta- 
tion consisted  of  the  bureau  chief's  sign-off  on  the  design 
document.  The  bureau  chief  indicated  there  is  some  review  per- 
formed but   it  is  not  documented. 
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Review  by  a  supervisory  person  can  be  very  productive. 
The  review  could  disclose  errors  in  work  or  faulty  assumptions 
which  can  affect  the  quality  of  a  system. 

The  analysts  should  review  the  work  of  the  programmers  to 
determine  the  analyst's  instructions  were  carried  out  properly. 
Documentation  of  the  programmer's  work  should  be  such  that  the 
analyst  can  easily  review  testing  procedures  for  propriety  and  can 
compare  program  documentation  and  test  results  to  the  analyst's 
instructions.      This  review  should  be  documented. 

For  the  analyst's  work,  the  systems  development  section 
supervisor  should  probably  perform  the  review.  This  review  will 
generally  be  less  detailed.  The  supervisor  should  be  concerned 
the  analyst  used  proper  assumptions,  and  conclusions  and  design 
are  supported  by  the  analyst's  work.  The  supervisor  may  also 
want  to  spot-check  the  analyst's  review  of  programmer  work. 
Again,   there  should  be  some  evidence  of  the  review. 

RECOMMENDATION   #10 

WE    RECOMMEND    A    SUPERVISORY    REVIEW    BE    CONDUCTED 

OF  ANALYST'S  AND   PROGRAMMER'S  WORK. 


Implementation   Planning 

A  well  controlled  development  process  usually  requires  a 
detailed,  step-by-step  implementation  plan  for  a  new  system.  The 
implementation  plan  is  a  vehicle  for  assuring  all  parties  are  aware 
of  their  responsibilities  and  for  assuring  all  procedures  are 
included. 

We  found  detailed  implementation  plans  were  not  prepared  for 
the  three  completed  systems  we  reviewed.  Two  of  the  systems  had 
a  general  outline.  The  third,  the  implementation  of  an  automated 
system  to  replace  a  manual  system,   had  no  implementation  plan. 

The  bureau  chief  stated  the  user  is  supposed  to  plan  imple- 
mentation and  SDB  does  not  place  a  priority  on  this  aspect.  In 
our  opinion  placing  this  responsibility  solely  with  the  agency  may 
not    be    appropriate.       The    agency    often    lacks    knowledge    of    the 


19 


internal  workings  of  the  system  and  potential  problems  which  can 
arise  during  implementation.  Therefore,  we  believe  SDB  should 
initiate  and  coordinate  implementation  planning.  Both  SDB  and  the 
agency  should  sign  the  implementation  plan  showing  their  concur- 
rence. 

Also,    SDB    should   define  what   elements   should   be   included    in 
the  plan.      Among   these  may  be: 

1.  A  chronological  order  of  events  and  who  is  to  perform  each. 

2.  A  timetable  for  completing  the  tasks. 

3.  The  procedures  for  validating  the  results  of  the  new  system 
against  the  results  of  the  old  system   (EDP  or  manual). 

4.  The  procedures  for  converting  old  system  data  (manual  or 
EDP)  for  use  with  the  new  system  and  for  assuring  the 
conversion  is  accurate  and  complete. 

RECOMMENDATION   #11 

WE  RECOMMEND  SDB  INITIATE  IMPLEMENTATION  PLANNING 
AND  COORDINATE  SDB  AND  AGENCY  IMPLEMENTATION 
PLANNING   EFFORTS. 


Post-Implementation   Review 

The  last  part  of  a  successful  systems  development  process  is 
a  review  of  the  system  and  how  the  system  is  operating  six  months 
to  a  year  after  implementation.  Post-implementation  review  pro- 
vides feedback  concerning  what  was  successful  about  the  develop- 
ment process  and  where  the  development  process  needs  improve- 
ment. The  post-implementation  review  should  be  performed  by 
someone  outside  the  development  team.  Included  in  the  items  for 
analysis  should  be: 

1.  Whether  the  system  is  meeting  the  user's  needs. 

2.  Whether     the     estimated     costs     and     benefits     are     reasonably 
related  to  the  actual  costs  and  benefits. 

3.  How  much   the   user   had   to   pay   for  maintenance  and   enhance- 
ment to  get  the  system  to  run  as  originally  desired. 
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4,     What   procedures   were   used   in   the   development   process   which 
either  significantly  helped  or  hindered  the  effort. 


Currently,  SDB  does  not  have  a  post-implementation  review 
process  so  they  are  not  getting  formal  feedback  on  the  develop- 
ment process. 

SDB  has  established  a  tentative  schedule  for  implementing  a 
post-implementation   review  process. 

RECOMMENDATION   #12 

WE    RECOMMEND    SDB    IMPLEMENT    A    POST-IMPLEMENTATION 

REVIEW  PROCESS. 


CHAPTER  V 

SYSTEM  MAINTENANCE 

After  developing  a  computer  system.  Systems  Development 
Bureau  (SDB)  may  enter  into  an  agreement  with  the  agency  to 
provide  assistance  concerning  problems  with  the  system.  The 
agreement  includes  provisions  for: 

— System  Consultation  -  Researching  and  answering  questions 
regarding  the  functioning  of  the  system.  Providing  assis- 
tance to  the  customer  in  troubleshooting  system  problems. 

—  Production  Recovery  -  Responding  to  abnormal  system  ter- 
minations. Restoring  or  recovering  system  files  or  programs 
to  operational  status.  Responding  to  suspected  system 
problems. 

— Minor  System  Maintenance/Enhancements  -  Required  system 
maintenance  that  does  not  have  significant  impact  on  the 
system.  Desired  system  enhancements  (modifications  or 
additions)  that  do  not  have  significant  impact  on  the  existing 
system  or  that  do  not  make  substantial  additions  to  the  sys- 
tem. 


The  request  for  assistance  may  originate  via  a  System  Sup- 
port/Enhancement Request  form  or  a  telephone  call.  Only  system 
maintenance/enhancements  requests  are  supposed  to  be  documented 
via  an  enhancement  request  form.  The  enhancement  request  form 
can  be  completed  by  SDB  or  by  the  agency.  If  the  agency  re- 
quests the  change  via  a  telephone  call,  the  enhancement  request 
form  may  not  be  signed  by  an  agency  official. 

The  enhancement  request  form  is  designed  to  document 
requested  modification  or  enhancements.  Upon  completion  of  the 
change,  the  modifications  made  are  noted  on  the  request  form.  A 
copy  is  maintained  by  SDB  and  a  copy  is  sent  to  the  agency. 

A  Program  Change  Request  (PCR)  form  is  also  supposed  to  be 
completed.  This  form  describes  the  change  in  more  technical  terms 
and  lists  the  programs  affected  by  the  change.  The  PCR  number 
is  to  be  listed  on  the  enhancement  form  for  easy   reference. 
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The  general  nature  of  the  changes  is  also  supposed  to  be 
listed  in  a  section  of  the  computer  program.  The  PCR  number  is 
to  be  referenced   in   this  section. 

If  a  change  needs  to  be  made  immediately  because  a  system 
does  not  operate  properly  during  processing,  SDB  completes  an 
emergency  change.  These  changes  do  not  require  any  documenta- 
tion as  support  or  as  evidence  of  the  work  performed. 

We  reviewed  the  documentation  pertaining  to  four  systems 
maintained  by  SDB.  We  noted  problems  concerning  documentation 
of  maintenance/enhancements  and  emergency  changes.  There 
concerns  are  discussed  below. 

DOCUMENTATION  OF  MAINTENANCE 

According  to  Computer  Control  and  Audit  (Mair,  Wood,  and 
Davis),  Control  Objectives  (EDP  Auditors  Foundation),  and  EDP 
Auditing  by  Auerbach,  a  well  designed  maintenance  process  would 
include  the  following  components: 

1.  All    requests    for    changes    would    be    initiated    by    the    agency 
with  written  approval  of  an  authorized  agency  official. 

2.  The    agency    would    be    given    an    estimate    of    the    cost    of    the 
change. 

3.  A    central    log    of   all    requests    would    be    maintained    to   assure 
timely  completion  of  all   requests. 

4.  The  programmer/analyst  making  changes  would: 

a.  Update  all  program  documentation   to  reflect  changes. 

b.  Note   in   a   section   of  a   program  or  module  when  and   what 
changes  were  made. 

c.  Cross-reference     the     request     for     changes     with     actual 
changes. 

d.  Changes    would    be    tested    according    to   established    stan- 
dards. 

5.  Changes     would     be     reviewed     to    determine    the     request    was 
completed   properly. 

6.  Change  documents   would   be  approved   by   SDB  and   the  agency 
before  being   placed   into  production. 
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The  following   sections  discuss  our  audit  findings. 

User's  Signature  on  Enhancement  Requests 

As  noted  previously,  a  System/Enhancement  Request  form  is 
supposed  to  be  completed  before  a  change  is  made  to  a  system.  If 
the  request  is  made  via  a  telephone  call,  the  user  often  does  not 
sign  the  request  form. 

EDP  Auditing  states  the  form  should  be  signed  by  the 
requesting  agency.  An  agency  official's  signature  on  the  mainte- 
nance form  authorizes  SDB  to  begin  work  on  the  project.  SDB 
and  the  agency  lack  assurance  changes  to  be  made  are  what  the 
agency  intended.  Without  the  signature,  SDB  lacks  support  to  bill 
the  agency  for  changes  they  made. 

RECOMMENDATION   #13 

WE  RECOMMEND  SDB  OBTAIN  AN  AUTHORIZED  OFFICIAL'S 
SIGNATURE  ON  THE  SYSTEM  SUPPORT/ENHANCEMENT 
REQUEST   FORMS. 

Update  of  Documentation 

Changes  to  programs  or  modules  are  often  not  reflected  in  the 
flowcharts  and  narratives  for  the  system.  SDB  relies  on  enhance- 
ment and  program  change  requests  to  document  system  changes. 
Enhancement  and  program  change  requests  often  do  not  detail  the 
problems,  solutions  to  the  problems,  and  specifications  for  the 
changes  needed.  In  one  case  we  examined,  the  documentation  was 
so  poor  it  required  a  line-by-line  search  of  hundreds  of  lines  of 
program  coding  to  determine  where  the  coding  changed  and  then  to 
evaluate  what  effect  the  change  had. 

Complete,  accurate,  and  up-to-date  documentation  provides 
the  ability  to  review  changes  for  propriety  and  reduces  the  depen- 
dence on  the  memory  of  the  person  who  did  the  work.  The 
current  lack  of  documentation  updates  makes  supervisory  review 
difficult.         Also,      cost      effective      maintenance      of      systems      is 
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jeopardized  because  of  the  time   it  would  take  a  new  programmer  or 
analyst  to  learn  about  a  system  given  current  documentation. 

We  recommend  SDB  require  documentation  to  be  updated 
whenever  changes  are  made  to  a  system.  This  should  include 
updates  to  system  and  program  documentation  in  sufficient  detail 
so  a  person  unfamiliar  with  the  the  program  could  assume  mainte- 
nance responsibility  with  minimal  difficulty. 

RECOMMENDATION   #U 

WE  RECOMMEND  SDB  REQUIRE  APPROPRIATE 

DOCUMENTATION    BE    UPDATED    WHEN    CHANGES    ARE    MADE 
TO  A  SYSTEM. 


Program  Change  Log 

Current  SDB  standards  require  a  lop  of  changes  to  be  main- 
tained within  a  section  of  each  program.  We  noted  one  system  in 
which  programs  have  no  log  even  though  changes  have  been  made. 
For  a  second  system  the  log  existed  but  did  not  include  all  the 
changes  made  to  the  programs.  Thus,  the  program  change  log 
policy  is  not  being  followed. 

We  believe  the  program  change  log  is  a  valuable  tool  for 
explaining  changes  which  have  been  made  to  a  program  and  for 
serving  as  a  link  between  enhancement  requests  and  detailed 
changes. 

RECOMMENDATION   #15 

WE     RECOMMEND     SDB     MANAGEMENT     ENFORCE     ITS     STAN- 
DARDS  FOR   PROGRAM  CHANGE   LOGS. 


Cross-referencing      Enhancement     Requests     and     Program     Change 
Requests 

As  mentioned  earlier,  when  an  enhancement  request  results  in 
a  change  to  a  program,  a  program  change  request  (PCR)  is 
supposed   to  be  generated.      Enhancement  requests  indicate  changes 
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to  be  made  to  programs  but  do  not  mention  the  specific  programs 
changed.  PCRs  indicate  what  program{s)  has  been  changed  as  a 
result  of  the  enhancement  request.  Changes  made  to  the  programs 
are  to  be  written  on  the  PCR  or  the  PCR  control  number  is  to  be 
referenced  on  the  enhancement  request.  We  found  this  does  not 
happen. 

Sixty-six  enhancement  requests  for  one  system  were 
reviewed.  We  found  seventeen  requests  specifically  mentioned 
changing  a  program,  yet  there  were  no  PCRs  referenced.  If  the 
PCRs  were  created,  they  could  not  be  traced  from  the  enhancement 
request  without  a  thorough  knowledge  of  the  system.  None  of  the 
PCRs  mentioned  enhancement  requests  so  the  information  could  not 
be  traced  backwards. 

Another  system  reviewed,  with  over  200  enhancement 
requests,  did  not  have  any  PCRs  mentioned  on  the  enhancement 
requests.  Some  enhancement  request  numbers  were  listed  on  some 
of  the  PCRs.  This  system  has  all  the  information  pertaining  to  the 
change  on  the  enhancement  requests.  Just  the  programs  affected 
by  the  change  are  listed  on  the  PCRs.  A  person  cannot  determine 
from  the  enhancement  request  what  programs  were  changed,  or 
what  changes  were  made  to  programs  listed  on  the  PCRs. 

SDB's  administrative  and  technical  manual  states  the  PCR 
number  should  be  placed  on  the  enhancement  request.  The 
enhancement  request  number  should  also  be  on  the  PCR  so  a 
person  can  trace  a  change  either  way.  Without  a  cross-reference 
between  enhancement  requests  and  PCRs,  it  is  difficult  (if  not 
impossible)  to  tell  what  enhancement  request  relates  to  what  PCR 
(and  vice  versa)   and  what  program(s)   is  affected  by  the  change. 

RECOMMENDATION  #16 
WE   RECOMMEND   SDB: 

A.  ENFORCE      THE      REQUIREMENT      FOR      PCR      CONTROL 
NUMBERS   TO   BE   PLACED  ON    ENHANCEMENT   REQUESTS. 

B.  ESTABLISH   AND   ENFORCE  A   STANDARD  THAT    REQUIRES 
ENHANCEMENT      REQUESTS      TO      BE      REFERENCED      ON 
PCRS. 
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Testing  of  Programs  or  Modules 

There  are  no  standards  providing  general  guidelines  for  the 
testing  of  changes  to  programs  or  modules  (groups  of  programs). 
The  amount  of  testing  conducted  after  changes  have  been  made  to 
a  program  or  module  depends  on  the  programmer/analyst.  Typi- 
cally only  the  changed  program  is  tested.  If  the  changed  program 
feeds  data  to  another  program,  the  programmer/analyst  decides  if 
any  programs  "down  stream"  (those  programs  that  accept,  as 
input,   the  output  from  the  modified  programs)   need  to  be  tested. 

Computer  Control  and  Audit  and  Control  Objectives  suggest 
adequate  programming  standards  should  include  testing  of  program 
changes.  Preferably,  the  entire  system  should  be  tested  rather 
than  the  individual  programs  or  modules  which  were  modified.  At 
a  minimum,  the  test  should  include  the  system  that  is  "down 
stream."  If  testing  of  program  modifications  is  not  applied  to  the 
complete  system,  the  probability  of  the  introduction  of  errors 
increases. 

Bureau  management  indicated  full  system  tests  are  not 
practical  or  necessary.  We  agree  a  full  system  test  should  not  be 
conducted  after  all  changes  since  this  may  not  be  cost  effective. 
However,  we  believe  a  systems  test  should  be  considered  when  a 
major  change  has  been  made  or  after  a  large  number  of  small 
changes  have  been  made. 

RECOMMENDATION   #17 

WE  RECOMMEND  STANDARDS  FOR  TESTING  CHANGES  BE 
IMPLEMENTED,  INCLUDING  STANDARDS  FOR  TESTING  THE 
SYSTEM   "DOWN   STREAM"   FROM  THE  CHANGE. 


Review  of  Changes 

An  independent  or  supervisory  review  is  a  major  control  for 
assuring  work  is  performed  completely  and  accurately  and  stan- 
dards are  followed.  Currently,  there  is  no  review  of  the  mainte- 
nance activity  of  SDB   staff. 
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Our  concerns  about  lack  of  update  of  documentation  and 
non-compliance  with  SDB  standards  might  have  been  detected  and 
addressed  if  a  review  process  had  been  in  place.  Also,  agencies 
would  be  provided  with  some  assurance  only  authorized  changes 
are  made  and  are  made  completely  and  accurately.  In  our  opinion, 
agencies  currently  have  little  assurance  in  this  area. 

RECOMMENDATION   #18 

WE    RECOMMEND   SDB    IMPLEMENT  A   SYSTEM   FOR    REVIEWING 

MAINTENANCE  ACTIVITY. 

EMERGENCY   CHANCES 

During  the  running  of  a  system  (processing),  "bugs"  show 
up  which  are  not  detected  during  testing.  These  cases  usually 
require  an  emergency  change  to  the  Job  Control  Language  (JCL, 
basic  instructions  to  the  computer  telling  it  what  to  do,  i.e.,  what 
files  to  access,  what  program  to  use)  or  programs  to  allow  pro- 
cessing to  be  completed. 

SDB  officials  indicated  emergency  changes  are  controlled 
similarly  to  routine  maintenance  and  enhancements.  We  reviewed 
the  procedures  for  emergency  changes  to  programs  or  systems  and 
found  significant  non-compliance  with  controls.  The  following 
sections  discuss  our  recommendations  for  integrity  improvements 
relating  to  authorization,  change  procedures  for  JCL  and  pro- 
grams,  documentation,   and  review  of  application   results. 

Authorization 

Emergency  changes  are  requested  verbally  since  there  is 
insufficient  time  to  process  a  regular  written  request.  We  did  not 
find  any  written  follow-up  authorization. 

Written  follow-up  authorization  serves  several  purposes, 
including: 

— Providing    SDB    with    documentation    that    the    change    was    au- 
thorized. 
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— Showing    the   change   was   approved    by   an    appropriate   agency 
official. 

— Providing  a  trail  of  all  changes  made  to  a  program. 

— Providing  written  support  for  billings  and  payments. 

RECOMMENDATION   #19 

WE       RECOMMEND       SDB       REQUEST       WRITTEN       FOLLOW-UP 

AUTHORIZATION   FOR   EMERGENCY   CHANCES. 


Change  Procedures 

When  the  programmer/analyst  makes  emergency  changes,  the 
JCL  or  program  is  changed.  The  change  then  remains  part  of  the 
JCL  or  program.  Emergency  changes  are  not  tested  prior  to 
implementation  because  of  legitimate  time  constraints.  SDB  does 
not  require  testing  of  emergency  changes  after  implementation 
because  they  assume  running  would  disclose  problems. 

According  to  Computer  Control  and  Audit,  the  emergency 
nature  of  the  changes  increases  the  likelihood  of  errors  being 
introduced  into  the  system.  In  some  cases,  the  production  run  is 
not  a  satisfactory  substitute  for  regular  testing  because  the  pro- 
duction run  may  not  expose  the  system  to  as  many  possible 
combinations  of  transactions  as  a  test  would. 

We  believe  the  following  procedures  for  emergency  changes 
would  provide  better  system  integrity.  A  copy  of  the  system 
software  should  be  made,  changes  should  be  made  to  the 
copy,  and  the  copy  should  be  used  for  the  running  of  the 
computer  system.  After  the  emergency,  appropriate  test 
procedures  should  be  conducted  to  ensure  the  change  achieves  the 
desired  results.  The  current  software  should  then  be  replaced 
with  the  changed  copy. 

RECOMMENDATION   #20 

WE      RECOMMEND     SDB      REPLACE     PRODUCTION     SOFTWARE 

ONLY  AFTER  APPROPRIATE  TESTING. 
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Documentation 

We  found  instances  in  which  documentation  concerning  emer- 
gency changes  should  have  been  but  was  not  created.  If  a  major 
system  change  is  disclosed  during  an  emergency  change,  a 
program  request  form  will  be  generated,  but  this  is  the  only  type 
of  documentation  created. 

Documentation  of  maintenance  activity  is  necessary  to  provide 
for  ease  of  supervisory  review  and  for  cases  of  future  mainte- 
nance. Emergency  changes  should  be  no  different  from  regular 
maintenance  in  this   respect. 

We  recommend  emergency  changes  be  completely  documented. 
This  documentation  should  include  updates  to  system  and  program 
documentation  in  sufficient  detail  so  a  person  unfamiliar  with  the 
program  could  assume  maintenance  responsibility  with  minimal 
difficulty. 

RECOMMENDATION   #21 

WE    RECOMMEND    NECESSARY    DOCUMENTATION    BE    CREATED 

AFTER  AN   EMERGENCY   CHANGE. 


Review  of  Application   Results 

When  emergency  changes  are  necessary,  testing  of  the 
changes  is  usually  not  performed.  Instead,  the  system  is  run  for 
actual  processing  of  the  data.  This  procedure  is  performed  until 
the  system  appears  to  have  run  properly. 

Changes  made  using  this  emergency  procedure  provide  the 
agency  with  little  assurance  the  results  are  complete  and  accurate. 
The  reliability  of  the  results  can  only  be  determined  by  performing 
reasonableness  checks  or  reconciliations. 

Some  of  the  agencies  SDB  deals  with  are  not  sophisticated 
data  processing  users.  Currently,  SDB  does  not  inform  the 
agencies  of  the  inherent  weaknesses  in  emergency  change  proce- 
dures and  of  the  need  to  test  the  results  of  such  processing. 
Thus,  we  believe  SDB  should  encourage  agencies  to  test  the 
results  of  processing  when  emergency  changes  are  made. 
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RECOMMENDATION   #22 

WE    RECOMMEND    SDB    ENCOURAGE    AGENCIES    TO    TEST    THE 

RESULTS   OF   PROCESSING   WHEN    EMERGENCY    CHANGES   ARE 

MADE. 
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CHAPTER  VI 


SDB   CONTRACTS 


The  Systems  Development  Bureau  enters  into  two  types  of 
contracts.  It  enters  into  agreements  with  agencies  to  develop  or 
maintain  software.  SDB  also  contracts  with  consultants  for  some 
development  work.      We  examined  both  types  of  contracts. 

AGENCY  AGREEMENTS 

Systems  Development  Bureau  (SDB)  enters  into  an  agreement 
with  the  agency  before  developing  a  system.  The  agreement  runs 
for  the  duration  of  the  project.  An  agreement  is  also  signed  for 
maintenance/enhancement  worl<  requested.  The  maintenance  agree- 
ments are  for  12  months.  All  agreements  are  for  time  and 
materials  (SDB  staff  time  and  computer  charges).  We  found  weak- 
nesses with  the  provisions  in  both  development  agreements  and 
maintenance  agreements.  We  also  have  suggestions  relating  to 
payment  provisions. 

System  Development  Agreements 

Computer  Control  and  Audit  suggests  the  roles  and  respon- 
sibilities of  the  development  group  and  the  agency  be  spelled  out 
in  writing.  We  reviewed  System  Development  Bureau/agency 
agreements  and  found  the  agreements  are  not  specific  enough 
concerning  roles  and  responsibilities.  SDB's  agreements  did  not 
contain   the  following   information: 

1.  Performance  -  System  software  tests  are  performed  at  the 
agency  site  with  the  agency's  data  to  determine  whether:  a) 
the  products  meet  specifications  and  standards  described  in 
the  contract;  and  b)  they  perform  repetitively  on  a  variety  of 
data  without  failure.  Currently  the  criteria  which  a  system 
must  meet  to  be  accepted  is  not  spelled  out. 

2.  Agency  Assistance  (Responsibilities)  -  Agency  acknowledges 
performance  by  the  contractor  under  the  agreement  requires 
information  and  cooperation  from  the  agency.  The  agency 
must  provide  complete,  timely,  and  accurate  information  and 
all  other  data  and  information  necessary  including  the  Accep- 
tance   Test    Plan.      The    developer    is    not    responsible    in    any 
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manner  for  any  delayed,  defective  or  otherwise  nonconforming 
services  which  are  caused  by  the  agency  supplying  untimely, 
inaccurate,  or  inadequate  information  or  materials.  The 
agency  provides  any  necessary  backup  procedures  and 
parallel  operations.  The  agency  will  furnish  all  computer 
time.  Such  a  provision  would  put  the  agency  on  notice  that 
it  must  be  a  partner  in  the  process. 

3.  Assignment  and  Subcontracting  -  The  contract  cannot  be 
assigned  or  work  subcontracted  without  the  written  consent  of 
the  agency.  Notification  of  assignment  allows  the  agency  to 
contract  directly  with  the  proposed  subcontractor  if  the 
agency  believes  it  is  in  their  best  interests.  Also,  the 
agency  can  veto  the  assignment  if  they  have  good  cause  to 
object  to  the  subcontractor. 

4.  Warranties  -  Contractor  warrants  software  delivered  will  meet 
functional  and  performance  specifications.  Correction  of 
defects  are  free  of  charge  for  a  warranty  period  after  accep- 
tance by  the  agency.  Currently  agencies  must  pay  for  any 
and  all  defects  which   show  up  in   system  operation. 

5.  Modification  -  Agreement  constitutes  entire  agreement  and  no 
amendment  is  effective  unless  it  is  in  writing  and  signed  by 
both  parties.      This  would  formalize  modification  procedures. 

6.  Termination  -  Either  party  may  terminate  the  agreement  with 
written   notice.     This  would  formalize  termination  procedures. 

7.  Liaison  -  A  liaison  for  the  agency  is  not  designated.  This 
would  formalize  SDB  contact  point  and  reduce  the  possibility 
of  confusion. 


We  noted  a  concern  by  SDB  officials  that  their  agreements  are 
different  than  a  private  company  contract  in  some  respects. 
However,  the  principles  concerning  specifying  roles  and  respon- 
sibilities should  still  apply. 

RECOMMENDATION   #23 

WE  RECOMMEND  SDB  REVISE  ITS  AGREEMENTS  TO  SPECIFY 
IN  MORE  DETAIL  THE  ROLES  AND  RESPONSIBILITIES  OF 
THE   PARTIES. 
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System  Maintenance  Agreements 

We  also  reviewed  the  systems  maintenance  agreements  SDB 
enters  into  with  agencies  to  determine  whether  the  agreements  are 
reasonable  and  both  parties  are  adequately  protected.  We  found 
the  same  items  were  lacking  as  in  the  development  agreements, 
including: 

1 .  Performance 

2.  Warranties 

3.  Termination 
U.  Liaison 

As  with  the  development  agreements,  roles  and  responsibilities 
should  be  clearly  detailed  in  a  written  format.  Therefore,  we 
suggest  SDB  revise  its  maintenance  agreements  to  specify  in  more 
detail  the  roles  and   responsibilities  of  the  parties. 

RECOMMENDATION   #24 


WE  RECOMMEND  SDB  REVISE  ITS  MAINTENANCE  AGREE- 
MENTS TO  SPECIFY  IN  MORE  DETAIL  THE  ROLES  AND 
RESPONSIBILITIES  OF  THE   PARTIES. 


CONSULTING  SERVICES 

Systems  Development  Bureau  occasionally  enters  into  contracts 
for  outside  consulting  services  if  the  bureau  cannot  provide  the 
work  necessary  to  an  agency.  It  has  entered  into  two  such 
contracts  since  1982.  One  was  for  a  person  to  write  a  program  for 
an  existing  system.  The  other  was  for  two  people  to  do  analy- 
sis/programming work  for  various  projects.  We  reviewed  the 
management  controls  concerning  the  monitoring  of  the  work  pro- 
duced by  the  consultants.  We  found  the  work  done  by  the  consul- 
tants is  not  monitored  any  more  than  the  work  done  by  the 
in-house    staff    (see    Chapter   III).      The    work   done    by    consultants 
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should  be  monitored  and  reviewed  more  closely  than   staff  to  assui 
the  consultant  is  fulfilling  the  contract. 


RECOMMENDATION  #25 

WE  RECOMMEND  SDB  IMPLEMENT  REVIEW  PROCEDURES  FOR 

WORK  DONE  BY  CONSULTANTS. 
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CHAPTER  VII 


FUNDING 


During  the  1983  Legislative  session,  the  Systems  Development 
Bureau  (SDB)  was  included  in  the  Information  Systems  Division's 
internal  service  fund,  but  was  given  the  goal  of  being  self- 
supporting,  ISD  was  requested  to  report  SDB's  financial  position 
to  the  Legislative  Finance  Committee  on  a  monthly  basis.  SDB 
(and  its  forerunner.  Information  Systems  Division)  had  shown 
losses  in  fiscal  years  1981-82  and  1982-83  (see  page  6).  We 
reviewed  SDB's  income  and  expenses  for  fiscal  year  1983'-84  to 
determine  if  SDB  was  self-supporting  during  that  time  period. 
Our  findings  are  discussed   below. 

ALLOCATION  OF  COSTS 

Currently  SDB  does  not  pay  any  of  Information  Services 
Division's  (formerly  Computer  Services  Division)  overhead 
expenses.  In  fiscal  year  1983-84  ISD's  overhead  was  $244,059.  If 
SDB's  proportionate  share  had  been  ten  percent,  SDB's  share  of 
the  overhead  would  have  been  $24,000.  If  this  expense  had  been 
included  in  the  determination  of  the  hourly  billing  rate  for  fiscal 
year  1984,  the  billing  rate  of  $29  per  hour  would  have  increased 
$1. 

SDB  includes  computer  processing  charges  incurred  by  SDB 
but  not  billed  to  SDB's  customers  in  its  billing  rate.  However, 
the  computer  charges  are  not  included  in  SDB's  operating  state- 
ments. For  fiscal  year  1983-84,  this  amounted  to  about  $36,000  or 
about  six  percent  of  SDB's  expenses. 

Illustration  3  on  the  following  page  shows  that  inclusion  of 
these  two  expense  items  would  have  reduced  SDB's  surplus  by 
$60,000.     The  bureau  would  have  recorded  a  small  deficit. 

Since  the  Legislature  wanted  to  monitor  SDB's  financial  status 
and  it  is  relying  on  Information  Services  Division's  reports,  we 
suggest  ISD  include  computer  charges  and  a  portion  of  its 
overhead  expenses  to  more  correctly  reflect  SDB's  financial 
position. 
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RECOMMENDATION   #26 

WE  RECOMMEND  ISD  INCLUDE  A  PORTION  OF  ISD  OVER- 
HEAD AND  INCLUDE  SDB'S  COMPUTER  CHARGES  WHEN 
REPORTING   SDB'S   FINANCIAL  POSITION. 


INCOME  AND   EXPENSES  COMPARISON 

We    looked    at    SDB's     income    and    expenses     for     fiscal     year 

1983-84     to    determine     if    SDB    was    self-supporting  for    that    time 

period.      The  following   illustration  details  the  income  and  expenses. 

INCOME  AND   EXPENSES    (Unaudited) 
Fiscal  Year   1983-84 

Income  $657,189   } 

Expenses  606,531   ^ 

Allocated   ISD  Overhead   *  24,406   ^ 

Computer  Processing  Charges   **  34,860 

Profit    (Loss)  $    (8,608) 


*  Since  SDB  is  supposed  to  be  self-supporting  SDB's  proportionate 
share  of  ISO's  overhead  was  deducted,  even  though  SDB  is 
presently  not  billed   for   this   expense   (see  Recommendation   //26)  . 

**  These  are  charges  to  computer  time  that  cannot  be  billed  to  an 
agency  for  which  SDB  is  developing  a  computer  system.  These 
charges  are  not  billed  to  SDB  but  are  paid  by  ISD.  We  included 
them  because  SDB  is  supposed  to  be  self-supporting  and  all 
costs   should  be   included. 


Source:      Statewide   Budget   and  Accounting   System 

2 

Source:   Office  of  the  Legislative  Auditor  from  ISD  Records 

Source:   Systems  Development  Bureau 

Illustration  4 

Our  analysis  indicates  that  SDB  had  a  small  operating  loss  in 
fiscal  year  1983-84,  The  primary  reason  for  the  loss  is  that  ISO's 
overhead  was  not  considered  when  figuring  the  billing  rate  for 
services. 
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AGENCY   RESPONSE 


DEPARTMENT  OF  ADMINISTRATION 

DIRECTOR'S  OFFICE 


TED  SCHWINDEN,  GOVERNOR  MITCHELL  BUILDING 


STATE  OF  MC>JTANA' 


(406)  444-2032  i        HELENA,  MONTANA  59620 


August  21,  1984 


R^f^n-!\/cr\ 


Robert  R.  Ringwood  ■   ^  J  i^ ^4 

Legislative  Auditor 

State  Capitol,  Room  135  '^''■■''' '"'-''  'tOSIATiVE  AUDITnp 

Helena,  Montana  59620  ""^ 

Dear  Mr.  Ringwood: 

We  have  reviewed  the  recommendations  on  the  EDP  Audit  of  the  Systems 
Development  Bureau. 

We  have  concurred  with  most  of  your  recommendations.  In  most  cases  we  feel 
we  are  already  complying  with  the  basic  intent  of  the  recommendation  but 
acknowledge  that  we  need  to  improve  the  means  we  use  to  formalize  the 
process,  insure  good  communication  with  the  agency  receiving  the 
programming  services  and  provide  better  auditability  of  services  performed. 
We  believe  that  most  of  the  recommendations  identify  areas  where 
improvements  would  aid  in  the  complicated  and  difficult  process  of 
providing  computer  programming  services  for  State  agencies. 

Our  responses  to  each  of  the  recommendations  are  enclosed.  We  appreciate 
the  opportunity  we  have  had  to  interact  with  your  staff  on  these  issues  and 
also  the  opportunity  provided  to  us  at  this  time  to  respond  formally  to  the 
recommendations . 

Sincerely, 


MORRIS  L.  BRUSETT,  Director 
Department  of  Administration 


Enclosure 
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'  EQUAL  OPPORTUNITY  EMPiOYER 


.f 


RESPONSE  TO  THF  '  ,  '  UDIT  OF 

THE  SYSTEMS  DF'  _bOr.iENT  BUREAU 

DATE:   08/21/84 


CHAPTER  III  -  MANAGEMENT  CONTROLS 

D  OF  A  RESPONSE  TO  RECOMMENDATION  in 

We  concur.  SDB  has  identified  several  areas  to  review  as  a  result  of  the 
preliminary  audit  findings.  In  the  course  of  our  review  we  will  evaluate 
each  area  to  determine  if  adequate  management  controls  exist  and  if 
controls  can  be  verified  through  documentation.  Vhere  appropriate,  we  will 
implement  additional  management  controls. 

CHAPTER  IV  -  SYSTEMS  DEVELOPMENT  PROCESS 

Control  Descriptions  in  Design  Documents 

D  OF  A  RESPONSE  TO  RECOMMENDATION  //2 

We  do  not  concur.  Although  we  agree  that  control  must  be  incorporated  in 
design  documentation,  we  do  not  agree  that  all  of  the  systems  are 
inadequate  in  the  areas  mentioned.  Also,  lack  of  clear  discussion  of  some 
of  these  controls  does  not  mean  that  the  systems  reviewed,  or  other  systems 
designed  by  this  Bureau  have  inadequate  controls.  In  fact,  control  is  an 
inherent  aspect  of  design.  For  example,  detective,  preventive,  and 
corrective  controls  are  not  specifically  noted  as  such  in  design 
documentation,  but  are  provided  for  in  documentation  -of  functional 
requirements  for  edit  and  update  criteria  and  reporting  requirements. 
Likewise,  transaction  trails  may  not  be  readily  identified  as  such  because 
of  the  report  or  file  names  assigned.  In  summary,  we  believe  that  these 
issues  are  currently  provided  for  in  SDB's  system  design  and  documentation. 

Discussion  beyond  what  is  presently  done  in  the  functional  design  would  not 
assure  adequate  security  and  integrity  of  the  system.  It  is  important  to 
assure  that  the  design  provides  the  appropriate  level  of  control.  This 
assurance  can  best  be  accomplished  by: 

Insuring  Systems  Analysts  and  Programmer/Analysts  responsible  for 
design  know  and  understand  control  requirements  and  how  they  are 
satisfied  in  the  design. 

Insure  that  the  approp.  xate  control  documentation  is  documented  at 
the  appropriate  time. 

Methods,  procedures  and  standards  will  be  reviewed  and  revised  during  the 
next  year  to  improve  the  Bureau's  ability  to  assure  appropriate  control  is 
provided  in  all  systems  designed  and  developed  by  the  Bureau. 
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Design  Alternatives 

D  OF  A  RESPONSE  TO  RECOMMENDATION  #3 

We  do  not  concur.  The  systeni(s)  designed  by  the  SDB  must  be  designed  to 
meet  the  functional  requirements  of  the  requesting  state  agency.  The  entire 
design  process  involves  user  agency  and  analyst  discussion  of  major  and 
minor  design  alternatives  and  options.  The  result  being  functional 
specifications  and  the  proposed  technical  solution.  If  the  requesting 
agency  management  decides  that  the  cost  of  implementing  and  operating  the 
appropriate  technological  solution  is  not  cost  beneficial,  then  SDB  and 
agency  management  need  to  cooperatively  determine  which  of  the  alternatives 
to  analyze.  These  alternatives  involve  planning,  reduction  in  scope,  and 
delay  in  implementation  of  lower  priority  requirements  as  well  as 
alternative  technological  solutions. 

We  do  agree  that  the  appropriate  consideration  of  alternatives  in  the 
design  process  is  a  critical  part  of  successful  systems  design.  Therefore, 
we  will  implement  the  following  items  to  ensure  design  alternatives  are 
appropriately  considered: 

Define  the  direction  of  a  project  as  early  as  possible  in  terms  of 
the  agreed  (between  SDB  and  agency  management)  approach  to  the 
project.  This  will  include  whether  the  scope  of  the  project  is  to 
define  the  functional  requirements  and  the  technological  solution, 
or  whether  the  scope  is  limited  by  factors  such  as  cost,  time, 
future  growth  or  life  expectancy  of  the  system. 

Emphasize  the  importance  of  appropriate  consideration  of  design 
alternatives  in  developing  functional  specifications  to  the  SDB 
staff. 

Develop  an  outline  of  the  various  alternatives  available  in 
systems  design  with  advantages,  disadvantages  and  factors  to 
consider  in  each  option. 

Incorporate  in  a  management  review  criteria  designed  to  verify  the 
systems  analyst  considered  the  various  alternatives  in  arriving  at 
the  final  design  proposal. 

In  the  future  you  should  expect  to  see  a  shift  in  technical  design 
decisions  in  favor  of  "state-of-the-art"  because  of  the  rapid  improvements 
in  software  productivity  tools  for  data  processing  professionals  and  the 
decreases  in  hardware  costs.  Designing  system  using  1970 's  technology  will 
result  in  systems  that  will  be  obsolete  in  the  1980 's  and  will  perpetuate 
the  problems  the  D.P.  industry  currently  faces  (i.e.,  cost,  quality, 
backlog,  delays,  etc.). 
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Quantification  of  Benefits 

D  OF  A  RESPONSE  TO  RECOMMENDATION  #4 

We  concur.  Whenever  possible  benefits  will  be  quantified  when  cost/benefit 
projections  are  being  prepared  for  major  systems  development  projects.  We 
intend  to  accomplish  this  by  the  following: 

Insuring  at  the  outset  of  major  projects  that  the  parameters  and 
responsibilities  for  cost/benefit  projections  and  analyses  have 
been  defined  with  participation  by  user  agency  management. 

Increase  staff  awareness  of  the  importance  of  quantifying 
benefits.  This  will  be  accomplished  through  systems  analysis 
training  that  includes  cost  benefit  analysis  and  analyst  staff 
meetings  with  cost  benefit  analysis  as  a  topic. 

Establish  management  review  criteria  for  verifying  that  benefits 
have  been  appropriately  quantified. 

We  think  it  is  misleading  to  state  that  benefits  are  not  quantified.  SDB 
has  always  quantified  benefits.  Based  on  past  discussions  with  the  audit 
staff,  we  feel  the  issue  is  one  of  disagreement  over  which  cost  categories 
can  be  quantified,  not  that  we  did  not  quantify  the  benefits.  We  feel  we 
quantify  all  benefits  that  can  be  predicted  with  a  reasonable  degree  of 
accuracy . 

Present  Value  Analysis 

D  Of  A  RESPONSE  TO  RECOMMENDATION  y/5 

We  do  not  concur.  SDB's  cost  benefit  analysis  of  a  proposed  system  provides 
the  agency  with  all  information  necessary  to  make  a  decision.  We  feel  the 
manner  in  which  an  agency  interprets  the  information  in  order  to  arrive  at 
a  decision  is  best  left  to  the  agency. 

We  do  not  concur  for  the  following  reasons: 

Net  present  value  analysis  ignores  the  unquantif iable  (intangible) 
benefits  of  the  system.  An  agency  may  have  valid  reasons  for 
weighting  the  intangible  benefits  higher  than  would  normally  be 
done.   A  net  present  value  analyis  inhib^'  s  this  approach. 

Selecting  a  required  j..^i.e  of  return  has  a  great  deal  of  influence 
on  the  analysis.  It  is  difficult  for  SDB  to  determine  which  rate 
should  be  used  to  most  appropriately  compute  the  project  benefit, 
given  each  agency's  unique  circumstances.  The  agency  is  in  the 
best  position  to  select  a  proposal  evaluation  strategy  that  best 
meets  iti,  needs. 
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Testing  Standards 

D  OF  A  RESPONSE  TO  RECOMMENDATION  #6 

We  concur.  During  this  fiscal  year  we  will  review  and  revise  the 
Administrative  and  Technical  Manual.  Testing  standards  and  guidelines  will 
be  one  of  the  topics  covered.  The  standards  and  guidelines  will  be 
developed  based  on  our  analysis  of  the  appropriate  and  practical  level  of 
responsibility  for  preparation  and  review  of  both  plans  and  results. 
Therefore,  the  standards  may  not  specify  the  same  definition  of 
responsibility  as  implied  in  your  discussion. 

Systems/Acceptance  Test 

D  OF  A  RESPONSE  TO  RECOMMENDATION  //7 

We  concur.  However,  the  recommendation  could  be  interpreted  to  mean  that 
we  have  not  tested  systems.  We  must  clarify  this  possible 
misinterpretation  by  stating  that  we  have  always  tested  systems,  but  the 
quality  of  the  testing  was  not  always  as  good  as  it  should  be. 

You  have  recommended  implementation  of  two  procedures  to  improve  quality. 
We  do  not  agree  that  these  procedures  would  significantly  improve  the 
quality  of  testing  in  the  SDB  environment.  There  are  too  many  factors  to 
consider  in  planning  the  systems/acceptance  tests.  Implementation  of  these 
procedures  would  result  in  an  administrative  burden  in  justifying  and 
documenting  the  reasons  that  the  procedures  were  not  followed. 

We  believe  that  implementation  of  testing  standards  (RECOMMENDATION  #6) 
will  improve  testing.  In  addition,  better  system  test  and  implementation 
planning  by  both  SDB  and  the  user  agency  will  result  in  better  quality. 

Better  planning  can  be  achieved  by  the  following: 

insuring  that  user  agencies  understand  their  responsibilities  in 
testing  and  the  commitment  that  will  be  required  to  test  the 
system. 

insuring  that  systems  analysts  have  the  appropriate  training  and 
guidelines. 

The  review  and  update  of  the  Administrative  and  Technical  Guide  will 
address  these  issues.  Analyst  training  will  provide  through  staff  analyst 
meetings  and  other  technical  training,  if  available.  Continued  effort  in 
improving  communications  between  SDB  and  the  user  agency  should  improve  the 
understanding  of  the  level  of  commitment  and  responsibility  required  to 
successfully  test  a  system. 
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Documentation  Standards 

D  OF  A  RESPONSE  TO  RECOMMENDATION  #8 

We  concur.  During  the  review  and  update  of  the  Administration  Technical 
Manual,  existing  documentation  standards  will  be  reviewed  and  additional 
standards  or  guidelines  will  be  developed,  as  deemed  appropriate. 

Completion  of  Documentation 

D  OF  A  RESPONSE  TO  RECOMMENDATION  #9 

We  concur.  Agreement  should  not  be  interpreted  to  mean  that  we  do  not  do 
this  now,  but  that  we  will  implement  procedures  to  improve  our  ability  to 
complete  documentation  in  a  timely  manner. 

We  do  not  totally  agree  with  your  recommended  steps.  Specifically, 
completion  and  review  of  documentation  in  the  testing  process  will  not 
assure  timely  completion  of  flow  charting  or  program  logic  narratives. 
This  documentation  must  be  complete  before  programs  are  written. 

Therefore,  we  will  implement  the  following  steps: 

1.  Insure  that  system  flow  charts  and  program  specifications  are 
complete  prior  to  assigning  dependent  tasks  by  management  review 
of  project  estimates  and  schedules. 

2.  Insure  that  system  flow  charts  are  updated  to  reflect  the 
installed  system  by  insuring  that  standards  and  guidelines  specify 
update  of  flow  charts  as  part  of  the  testing  task  and  by 
implementing  quality  assurance  processes.  We  will  be  evaluating 
computer  graphics  for  flow  charting. 

3.  Insure  that  operating  instructions  for  the  user  agency  are 
completed  by: 

emphasizing  to  the  responsible  user  team  members  the 
importance  of  the  user's  manual  and  encouraging  development 
and  maintenance  of  a  plan  and  schedule  for  its  completion. 

providing  ongoing  review  ^nd  critique  of  the  material 
produced  by  the  user  for  th*^  manual. 

providing  assist '-,.;e  vhen  necessary  by  drafting  a  portion  of 
the  manual. 

providing  a  draft  of  operating  instructions  that  incorporates 
the  technical  considerations. 

4.  Implement  a  method  for  documenting  exceptions  to  timely  completion 
of  documentation  and  insuring  subsequent  follow-up  and  completion 
before  the  completion  of  the  project. 
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Supervisory  Review 

D  OF  A  RESPONSE  TO  RECOMMENDATION  #10 

We  concur.  Our  agreement  should  not  be  interpreted  to  mean  that  there  is 
no  review  now,  but  that  we  concur  that  there  is  little  or  no  documented 
evidence  of  the  review. 

The  review  and  update  of  the  Administrative  and  Technical  Manual  will 
Include  a  review  of  the  systems  development  methodology  to  assure  that  the 
appropriate  review  points  and  review  responsibilities  are  documented.  The 
review  will  also  design  a  method  that  can  be  used  to  document  the  reviews. 

Until  formal  review  processes  are  more  clearly  identified,  we  will 
implement  the  following: 

the  new  Information  Services  Division  organization  provides  for 
more  supervisory  involvement  (Development  Project  Unit  Supervisor) 
in  projects. 

the  importance  of  review  and  walk  through  of  work  will  be  re- 
emphasized. 

Implementation  Planning 

D  OF  A  Comment 

Tne  report  states  that  the  Bureau  Chief  said  "SDB  does  not  place  a  priority 
on  this  aspect".  Although  minor,  this  is  a  misquote  that  changes  the 
intended  meaning  which  was  that  SDB  does  not  put  a  high  enough  priority 
(versus  no  priority)  on  monitoring  the  user  agency's  progress  in 
implementation  planning. 

D  OF  A  RESPONSE  TO  RECOMMENDATION  #11 

We  concur,  primarily  because  we  presently  do  this.  In  the  past,  the 
efforts  have  not  always  been  as  successful  as  desired  for  a  variety  of 
reasons;  including: 

schedule  and  budget  problems. 

lack  of  full  understanding  by  the  users  of  the  commitment 
required. 

Improved  implementation  planning  will  be  accomplished  by: 

insuring  the  Project  Manager  understands  and  concurs  with 
responsibility  for  the  plan. 

insuring  that  the  SDB  project  team  leader  participates  in 
preparation  of  the  plan. 
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insuring  that  the  SDB  proje'-'  ,  ..t  leader  monitors  progress  and 
takes  appropriate  alternative  dCi...on  if  progress  indicates  that 
the  plan  is  in  jeopardy. 

insuring  that  the  project  estimate  includes  sufficient  time  for 
the  appropriate  level  of  SDB  involvement  and  that  this  portion  of 
the  estimate  is  developed  in  cooperation  with  the  Project  Manager. 

modifying  the  Development  Methodology  to  consolidate  Conversion 
and  Implementation  Plans  to  one  plan,  the  Implementation  Plan, 
which  will  include  conversion. 

Post  Implementation  Review 

D  OF  A  RESPONSE  TO  RECOMMENDATION  #12 

We  concur.  During  the  audit  we  identified  a  tentative  schedule  for 
implementing  a  post -implementation  review  process.  That  schedule  will  be 
revised  during  planning  and  prioritizing  the  activities  necessary  to 
implement  all  the  recommendations  in  this  report. 

CHAPTER  V  -  SYSTEM  MAINTENANCE 

User's  Signature  on  Enhancement  Requests 

D  OF  A  RESPONSE  TO  RECOMMENDATION  #13 

We  concur  that  signatures  are  necessary  for  user  initiated  changes. 
However,  two  points  need  to  be  made  to  clarify  our  use  of  t'he  Enhancement 
Request  (ER) : 

SDB  does  have  authorization  to  change  a  system  and  bill  for  those 
services  via  a  system  support  service  agreement.  The  service 
agreement  is  the  vehicle  used  to  formally  establish  authorization 
and  bill  for  services,  not  the  ER.  The  service  agreement  provides 
a  practical  way  of  being  granted  authorization  in  a  manner  that  is 
sufficiently  flexible  and  administratively  efficient. 

Often  an  agency  is  not  in  a  position  to  know  that  a  change  is 
required  to  the  system  or  able  to  describe  the  change.  In  other 
cases,  the  nature  of  the  change  does  not  warrant  formal 
authorization.  In  these  case^ ,  it  ir  left  to  SDB  staff 
professional  judgment  to  deterr  ^ne  if  an  ER  is  required  and  if  it 
should  be  signed  by  tb-  agency.  If  SDB  staff  determines  the 
change  does  not  warrant  agency  authorization  but  determines  that 
an  ER  is  an  appropriate  means  to  document  the  change,  the  ER  will 
not  be  signed.  This  position  should  not  be  interpreted  as  a 
reluctance  on  our  part  to  document  agency  authorization.  We 
continually  encourage  agencies  to  document  their  change  requests 
via  the  ER  as  a  good  communications  tool  and  means  to  document 
agency  initiation  of  the  change. 
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update  of  Documentation 

D  OF  A  RESPONSE  TO  RECOMMENDATION  #14 

We  concur  to  the  extent  that: 

The  instructions  on  the  Enhancement  Request  form  will  be  changed 
to  place  even  greater  emphasis  on  the  user  completing  the  request 
in  terms  of  what  is  to  be  requested,  in  user  terms,  rather  than 
providing  the  request  in  technical  terms. 

Users  will  be  encouraged  to  complete  the  request  in  "what  needs  to 
be  done"  terminology  each  year  when  the  new  fiscal  year's  service 
agreements  are  distributed  and  any  new  systems  are  implemented. 

Program  Change  Requests  (PCRs)  will  document  the  change  in  terms 
that  clearly  indicate  to  the  reader  the  nature  of  the  change. 

Program  Change  Logs  will  be  maintained  in  each  program  that 
indicates  the  nature  of  the  change  and  the  originating  FOR. 

System  Flow  Charts  will  always  be  updated  to  reflect  each  change. 

Included  below  are  a  few  comments  regarding  points  made  that  we  disagree 
with: 

Detailed  documentation  in  the  Enhancement  Request,  Program  Change 
Request  and  source  code  have  proven  to  be  a  costly  and  misleading 
means  of  documenting  systems.  Detailed  comments  in  source  code 
add  substantially  to  the  time  it  takes  to  write  or  change  a 
program  and  it  adds  little  additional  value  due  to  the  readability 
of  the  high  level  languages  we  use,  i.e.,  COBOL.  However,  a 
general  description  of  a  module's  function  is  a  valuable 
documentation  aid  and  is  a  standard  in  all  programming.  A  general 
comment  avoids  the  danger  of  a  change  being  made  to  the  source 
code  but  not  being  made  to  the  comment.  A  general  comment  seldom 
requires  a  change. 

Cost  effectiveness  of  maintenance  cannot  be  measured  by  how  easily 
(or  if)  a  third  party  can  determine  what  affect  a  documented 
change  had  in  the  actual  program  code.  We  feel  the  critical 
factor  in  maintenance  effectiveness  is  assignment  of  a  programmer 
with  the  appropriate  level  of  experience  to  be  able  to  cost 
effectively  maintain  a  given  system.  There  is  no  practical  means 
of  providing  a  simple  reference  to  all  code  changes  as  a  result  of 
the  request,  regardless  of  who  it  is  that  is  reviewing  the  change. 

We  don't  ever  anticipate  producing  documentation  in  sufficient 
detail  so  a  person  "unfamiliar"  with  the  program  and  the  system 
could  assume  maintenance  responsibility  with  "minimal"  difficulty. 
We  do  not  feel  this  is  an  appropriate  standard  to  establish  in 
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judging  the  quality  of  document- "  ior. ;  in  fact,  we  question  whether 
any  amount  of  documentation  c^n  meet  this  standard. 

Program  Change  Log 

D  OF  A  RESPONSE  TO  RECOMMENDATION  #15 

We  concur.  The  program  change  log  standard  needs  to  be  enforced.  This 
item  will  be  emphasized  to  SDB  staff  and  compliance  verified  by  SDB 
management . 

Cross -Referencing  Enhancement  Requests  and  Program  Change  Requests 

D  OF  A  RESPONSE  TO  RECOMMENDATION  inS 

We  concur.   We  intend  to  do  the  following: 

Modify  the  Enhancement  Request  to  include  a  block  to  indicate  all 
programs  affected. 

Modify  the  Program  Change  Request  to  identify  all  Enhancement 
Requests  causing  the  program  to  be  changed  (often  program  changes 
are  grouped  together  from  several  Enhancement  Requests  to  reduce 
costs) . 

Update  the  Administrative  and  Technical  Guide  to  clearly  state  the 
cross-referencing  requirements . 

Emphasize  to  the  staff  the  importance  of  proper  cross-reference  of 
Enhancement  Requests  and  Program  Change  Requests. 

Testing  of  Programs  or  Modules 

D  OF  A  RESPONSE  TO  RECOMMENDATION  //1 7 

We  concur  that  general  guidelines  should  be  developed  for  testing  program 
changes . 

It  would  be  impossible  to  develop  detailed  testing  standards  that  would 
address  the  infinite  variety  of  situations  encountered  in  the  change 
environment.  It  is  the  responsibility  of  the  programmer/ analyst  to 
evaulate  the  unique  circumstances  in  a  given  situation  and  outline  a  test 
plan  that  will  adequately  test  the  change.  No  standards  can  replace  the 
skills  and  abilities  required  of  uhe  programmer/analyst.  Also,  it  seems  a 
misunderstanding  exists  about  how  SDB  conducts  a  test.  In  most  cases  it  is 
far  easier  (and  consistent  with  your  recommendation)  to  test  related  groups 
of  programs.  Data  fed  from  one  program  into  another  is  usually  much  more 
difficult  to  create  "by  hand"  rather  than  letting  the  preceding  program 
generate  the  data.  This  provides  the  verification  of  program  to  program 
transfer  that  is  critical.  However,  we  stop  short  of  agreeing  that  a  full 
systems  test  should  be  conducted.  Most  changes  do  not  impact  every  part  of 
the  system.   The  programmer/analyst's   responsibility  is  to  isolate  the 


47 


impact  of  the  change  to  affected  programs  and  insure  all  programs  within 
the  isolated  set  will  work  properly  after  the  change. 

Although  we  do  not  agree  that  full  systems  tests  are  practical  or  necessary 
we  do  feel  the  importance  of  proper  problem  isolation  and  testing  is  an 
important  concept  that  cannot  be  overemphasized.  We  also  include  this  as 
an  emphasis  item  in  future  training  and  staff  meetings. 

Review  of  Changes 

D  OF  A  RESPONSE  TO  RECOMMENDATION  #18 

We  concur.  Although  there  currently  is  some  review  of  maintenance 
activity,  there  is  not  a  formal  system  for  review.  This  system  should 
include: 

periodic  independent  or  supervisory  review  to  insure  that 
policies,  standards,  and  control  procedures  are  consistently 
followed. 

periodic  quality  assurance  to  assure  that  work  is  performed 
completely  and  accurately. 

re-emphasis  of  the  importance  of  the  team  approach  and  value  of 
walk  thru  and/or  review  of  the  actual  work  by  a  team  member  when 
practical  and  effective. 

a  method  for  documenting  the  review  process. 

Authorization 

D  OF  A  RESPONSE  TO  RECOMMENDATION  #19 

We  concur.  Refer  to  Recommendation  #13  for  conditional  comments. 
Emergency  change  and  enhancement  authorization  requirements  are  the  same. 

Change  Procedures 

D  OF  A  RESPONSE  TO  RECOMMENDATION  #20 

We  concur.  With  few  exceptions,  this  procedure  is  currently  in  place  and 
complied  with. 

However,  it  is  not  practical  or  reasonable  to  implement  procedures  that  do 
not  exercise  the  judgment  of  the  responsible  support  personnel.  JCL  and 
program  emergency  change  procedures  need  to  be  different.  The  infinite 
variety  of  production  recovery  problems  requiring  JCL  changes  make  setting 
a  standard  change  procedure  impractical.  We  will  emphasize  to  the  staff 
that  these  procedures  should  be  followed  for  production  recovery  JCL  and 
catalogue  procedures  changes,  when  practical.  Determination  of  the  best 
method  to  effect  the  change  will  continue  to  be  the  responsibility  of  the 
programmer/ analyst . 


Documentation 

D  OF  A  RESPONSE  TO  RECOMMENDATION  #21 

We  concur.  The  same  documentation  should  be  prepared  for  emergency  changes 
and  controlled  changes. 

Review  of  Application  Results 

D  OF  A  RESPONSE  TO  RECOMMENDATION  y/22 

We  concur.  SDB  has  always  and  will  continue  to  encourage  testing,  when  it 
is  appropriate.  Whether  or  not  testing  is  required,  and  how  extensive  the 
testing  should  be  must  be  discussed  with  the  responsible  user  agency. 

Users  should  be  fully  informed  regarding  the  status  of  the  system, 
including  a  thorough  description  of  problems  occurring  during  production 
processing  and  the  potential  impact  of  the  problem  and  problem  resolution. 
The  staff  normally  does  this.  However,  we  will  re-emphasize  to  the  staff 
that  they  should  approach  the  verification  issue  cooperatively  with  the 
user,  encouraging  the  user  to  monitor  system  processing  closely  after 
changes  are  made,  identifying  those  areas  that  were  affected  and  could 
potentially  fail,  and  encouraging  the  user  to  test  the  changes  if  there  is 
a  potential  for  impact  to  the  system's  integrity. 

CHAPTER  VI  -  SDB  CONTRACTS 

System  Development  Contracts 

D  OF  A  RESPONSE  TO  RECOMMENDATION  #23 

We  concur.  We  will  review  the  basic  format  used  for  development  agreements 
to  measure  how  well  they  adhere  to  key  principles  appropriate  to 
interagency  provision  of  services.   We  will  use  7  criteria: 

Identification  of  services 
Identification  of  responsibilities 
Identification  of  deliverables 
Identification  of  duration 
Identification  of  cost  or  cost  basis 
Identification  of  ownership 
Identification  of  control  of  changes 

We  feel  the  use  of  these  criteria  in  development  service  agreement  content 
will  ensure  adequate  protection  for  both  SDB  and  the  agency. 

In  the  past  a  service  agreement  addendum  was  attached  to  all  development 
service  agreements  listing  terms  and  conditions  associated  with  the  service 
agreement.  This  practice  was  discontinued  some  time  ago  because  it 
generated  more  controversy  than  it  resolved.  After  reviewing  your 
recommendation  and  original  "terms  and  conditions"  addendum  we  feel  it  is 
worthwhile  to  re-evaluate  the  use  of  addendum,   and  if  at  all  possible, 
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develop  an  addendum  that  will  improve  the  quality  of  the  service  agreement. 
We  will  initiate  an  administrative  project  to  evaluate  the  use  of  addendums 
and  identify  standard  service  agreement  requirements. 

We  have  specific  comments  about  the  items  you  indicated  were  not  in  SDB's 
agreements : 

1.  Performance.  We  feel  our  design  documents  adequately  detail  the 
criteria  which  a  system  must  meet  to  be  accepted.  Future 
agreements  will  formally  associate  the  design  document  with  the 
service  agreement  to  avoid  any  misunderstandings  regarding  what  is 
to  be  developed. 

4.  Warranties.  Blanket  warranties  will  not  be  provided.  In  the 
past,  attempts  to  provide  warranties  have  been  unsuccessful, 
primarily  because  it  is  very  difficult  to  determine  if  a  required 
change  is  the  result  of  SDB's  failure  to  deliver  or  the  agency  is 
requesting  a  change  from  the  original  design. 

However,  a  warranty  can  be  provided  that  obligates  SDB  to  deliver 
the  system  components  that  were  promised  in  the  service  agreement, 
e.g.,  system  flow  charts,  all  program  definitions.  A  warranty 
will  be  written  that  provides  this  kind  of  agency  protection. 

5.  Modification.  Modifications  are  adequately  covered  using  existing 
procedures.  Currently  a  Change  Request  is  used  to  document 
changes  of  sufficient  scope  to  warrant  formal  agency  request,  full 
specification,  cost  and  schedule  impact,  and  agency  approval.  A 
Change  Log  is  used  to  document  minor  design  changes  that  do  not 
warrant  the  analysis  of  a  formal  Change  Request. 

6.  Termination.  A  termination  clause  will  be  included  that  has  a 
time  limitation  to  protect  the  party  that  is  not  terminating  the 
contract.   A  30  day  notice  will  probably  be  used. 

7.  Liaison.  SDB  has  always  requested  the  agency  appoint  a  Project 
Manager.  This  practice  will  be  formalized  by  identifying  a 
liaison  in  the  service  agreement. 

System  Maintenance  Agreements 

D  OF  A  RESPONSE  TO  RECOMMENDATION  //24 

We  concur.   The  discussion  for  recommendation  #23  applies  here  also. 

Consulting  Services 

D  OF  A  RESPONSE  TO  RECOMMENDATION  #25 

We  concur  that  review  procedures  applicable  to  consultants  should  be 
implemented.  General  review  criteria  will  be  developed.  However,  we 
disagree  with  statements  suggesting  more  review  of  consultants  work  is 
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necessary  than  for  SDB  staff.  The  primn-  ;  r.  ason  for  hiring  a  consultant 
is  to  enable  more  work  to  be  accompi  -snea  without  diverting  SDB  attention 
away  from  the  projects  already  underway.  Requiring  additional  review  would 
defeat  the  purpose  of  hiring  the  consultants.  We  feel  the  consultant 
selection  process  is  the  means  used  to  verify  that  a  qualified  consultant 
will  be  assigned  to  the  project.  We  feel  the  contractor-consultant 
iolationship  must  be  structured  such  that  the  consultant  has  sufficient 
freedom  to  exercise  its  own  professional  judgment.  It  would  not  be  fair  to 
hold  them  liable  to  contract  performance  criteria  when  required  to  conduct 
a  project  according  to  SDB  procedures.  General  review  criteria  can  and 
should  be  developed  to  insure  compatibility  but  sufficient  flexibility  must 
be  provided  to  the  contractor  to  avoid  intefering  with  the  consultants' 
right  to  exercise  their  own  professional  judgment. 

CHAPTER  VII  -  FUNDING 

General  Comments 

ISD  Overhead  Expenses.  The  $244,059  you  use  as  the  basis  for 
overhead  allocation  includes  many  costs  which  are  not 
appropriately  included  in  SDB's  rate.  Here-are  a  few  examples: 
1)  $3,000  in  ISD  photocopier  fees  (SDB  provides  its  own 
photocopying),  2)  $2,000  in  office  supplies  (SDB  orders  and  pays 
for  its  own  office  supplies),  3)  $13,000  in  telephone  charges 
(SDB  is  already  charged  for  the  phones  installed  in  the  SDB 
offices),  4)  $8,000  in  remodeling  fees  (the  remodeling  feels 
were  not  for  the  SDB  offices).  The  primary  reason  for  allocating 
these  costs  to  the  administrative  center  was  due  to  the  difficulty 
in  accurately  assigning  these  charges  to  the  various  centers  that 
are  supported  by  the  computer  rate.  Most  of  the  charges  should 
not  be  allocated  to  SDB  as  they  are  not  supporting  SDB  operations. 
It  can  also  be  argued  that  allocating  any  of  the  ISD  overhead 
expenses  effectively  double  charges  SDB  for  SDB  overhead.  On  one 
hand  SDB,  like  all  other  agencies,  pays  for  computer  charges.  The 
computer  rate  includes  a  factor  necessary  to  recover  the  $244,059 
in  administrative  charges.  When  SDB  spent  $34,673  in  FY84  on 
computer  charges  it  financed  a  portion  of  the  ISD  administrative 
costs  in  the  same  way  all  other  users  of  the  computer  center 
financed  this  cost.  To  require  SDB  to  recovei  a  portion  of  the 
administrative  costs  as  suggested  would  douMe  charge  SDB.  It  can 
be  argued  that  SDB  makes  use  of  certain  ISD  administrative 
overhead  costs  more  than  other  user  agencies,  e.g.,  ISD  management 
and  staff  time  and  a  portion  of  this  costs  should  be  allocated  to 
SDB.  We  feel  it  is  mc-c;  appropriate  to  allocate  only  a  portion  of 
personal  services  expenses  back  to  SDB  to  avoid  SDB  being  double 
charged  for  operating  expenses  and  double  charged  through  the 
computer  processing  rate. 

SDB  Computer  Charges.  ISD  Year-To-Date  Billing  history  indicates 
total  FY«4  SDB  computer  charges  were  $34,673,  not  $36,272.  Actual 
charges  were  as  follows: 


Billing  Number 

YTD  Charges 

00-201 

$  9,735 

00-201 

13,392 

00-202 

4,673 

00-203 

16 

00-204 

6,572 

00-672 

285 

$34,673 

SDB  Computer  Charges  in  the  SDB  Rate.  The  reader  could  conclude 
from  reading  the  audit  that  SDB  does  not  pay  for  its  computer  use. 
The  hourly  rate  calculation  incorporates  SDB  computer  use  to 
properly  recover  the  charges  through  agency  billing.  This  has 
always  been  the  case,  even  though  the  charges  were  not  included 
when  the  financial  position  was  reported. 

Final  Income  and  Expense  Comparison.  Based  on  the  above  rationale 
we  feel  SDB's  financial  position  is  more  accurately  stated  as 
follows: 

Income  $657,189 

Expenses  612,591(1) 

Allocated  ISD  Overhead  8,660(2) 

Computer  Processing  Charges     34,673 

$1,265 

(l)includes  depreciation  recorded  in  responsibility  center  718 
(2)5%  of  Administrative  Support  personnel  charges 

D  OF  A  RESPONSE  TO  RECOMMENDATION  #26 

We  concur.   Specifically: 

We  will  include  5%  of  ISO's  administrative  personnel  charges  in 
reporting  SDB's  financial  position. 

SDB's  computer  charges  will  be  included  in  SDB's  financial 
reports . 
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